Papers with English captions
Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (D19-64)
Copied to clipboard
| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (D19-66)
Copied to clipboard
| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets have extensive labeled data for En-glish, but labeles are extremely scarce in other languages. |
| Approach: | They propose a method that leverages existing annotations with machine translation capabilities to create cross-modal language generation systems at web-scale. |
| Outcome: | The proposed model outperforms other candidates in evaluations performed over 5 target languages. |
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions. |
| Approach: | They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
| Outcome: | The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |